A rapidly convergent descent method for minimization

We are concerned in this paper with the general problem of finding an unrestricted local minimum of a function J[xu x2 • • •, xn) of several variables xx, x2, • •., xn. We suppose that the function of interest can be calculated at all points. It is convenient to group functions into two main classes according to whether the gradient vector g, = Wbx, is defined analytically at each point or must be estimated from the differences of values of/. The method described in this paper is applicable to the case of a defined gradient. For the other case a useful method and general discussion are given by Rosenbrock (1960). Methods using the gradient include the classical method of steepest descents (Courant, 1943; Curry, 1944; and Householder, 1953), Levenberg's modification of damped steepest descents (1944), a somewhat similar variation due to Booth (1957), the conjugate gradient method of Hestenes and Stiefel (1952), similar methods of Martin and Tee (1961), the "Partan" method of Shah, Buehler and Kempthorne (1961), and a method due to Powell (1962). In this paper we describe a powerful method with rapid convergence which is based upon a procedure described by Davidon (1959). Davidon's work has been little publicized, but in our opinion constitutes a considerable advance over current alternatives. We have made both a simplification by which certain orthogonality conditions which are important to the rate of attaining the solution are preserved, and also an improvement in the criterion of convergence.